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Background / Context: 

Description of prior research and its inteiiectuai context 

Charter schools are publicly funded schools that are given more freedom than traditional public 
schools to set their curricula, control staffing, and manage day to day operations. A growing 
body of evidence suggests that urban charter schools have the potential to generate impressive 
achievement gains for minority students living in high-poverty areas. A series of recent studies 
using admissions lotteries to identify causal effects reports large gains for charter schools located 
in Boston and other urban areas of Massachusetts, as well as the Harlem Children's Zone and 
Washington, DC (Abdulkadiroglu et al, 201 1; Angrist et al, 2010a, 2010b; Dobbie and Fryer, 
2011; Curto and Fryer, 2011). Studies of Chicago and New York charter schools also report 
positive effects (Hoxby and Rockoff, 2004; Hoxby, Murarka and Kang, 2009). 

While these results are encouraging, results for more diverse sets of charter schools are more 
mixed. A recent report evaluating roughly two dozen Massachusetts charter finds little evidence 
of achievement gains at schools outside of high-poverty urban areas (Angrist et al, 2011). Some 
of the estimates for non-urban Massachusetts charters show significant negative effects. These 
results echo findings from a multi-state study of 36 charter middle schools using admissions 
lotteries (Gleason et al, 2010). Here too, charter schools outside of urban areas seem to do little 
for achievement, though urban schools with high-minority, high-poverty enrollment generate 
some gains. The source of these striking differences in charter school effectiveness across 
settings is an open question. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

This study uses entrance lotteries to explore heterogeneity in the achievement effects of charter 
schools across demographic groups and between urban and non-urban areas in Massachusetts. 
We develop a framework for interpreting this heterogeneity using both student- and school-level 
explanatory variables. 

Setting: 

Description of the research iocation. 

Our sample includes 16 middle schools and six high schools from Massachusetts. Nine of the 
middle schools are in urban areas; seven are in Boston. The other seven are in non-urban areas. 
Four participating high schools are in Boston, and the other two are in non-urban areas. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

The set of applicants to charter schools in our lottery sample includes 6,214 middle school 
students and 4,207 high school students. Table 1 reports pre-lottery demographic characteristics 
for these students. Urban charter schools serve large fractions of minority students — 71 percent 
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and 87 percent of applicants are black or Hispanic in middle and high school, respectively. 

Urban applicants are also poor (more than 65 percent are eligible for subsidized lunch), and low- 
achieving; their pre-lottery test scores are more than 0.3 standard deviations below the state 
average in both math and English language arts (EL A). 

In contrast, non-urban charter schools serve a mostly white, high-achieving population. Less than 
three percent of non-urban applicants are black or Hispanic, and just over 10 percent are eligible 
for subsidized lunch. Non-urban students also enter with high baseline achievement levels; they 
score more than 0.3 standard deviations above the state average for each subject in middle 
school, and baseline achievement for non-urban high school applicants is even higher. The sharp 
disparities between the characteristics of urban and non-urban charter applicants provide useful 
variation for studying heterogeneity in the effects of charter schools. 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, inciuding detaiis of administration and duration. 

Educational practices differ dramatically across urban and non-urban charter schools. Table 2 
reports information from a survey of school administrators. Urban charter schools run a longer 
school day and year than do non-urban schools, and spend more time on math and reading 
instruction. Urban charters are also more likely to require parent and student contracts, are much 
more likely to use formal reward and punishment systems, and have slightly higher per-pupil 
expenditures. The survey reveals a sharp division between urban and non-urban charters with 
respect to the No Excuses approach to education. As discussed by Thernstrom and Thernstrom 
(2003) and Carter (2000), No principles include a strict disciplinary environment, an 

emphasis on student behavior and comportment, extended time in school, and an intensive focus 
on traditional reading and math skills. Seventy-one percent of urban charter administrators 
identify somewhat or fully with No ExCUSes, while no non-urban charter identifies with this 
approach. 

Research Design: 

Description of the research design. 

Our research design identifies the causal effects of attending charter schools by comparing 
students who win entrance lotteries to students who lose. Since these lotteries are random, any 
systematic difference between winners and losers must be due to the lottery outcome. This 
approach eliminates the selection bias that plagues observational studies of charter school 
effectiveness. 

Eormally, we estimate causal effects at charter schools with available lottery records using two- 
stage least squares (2SLS). The second-stage equation in this context is 

Yigt ~ ^2t fr l^2g fr ^^igt ^igt^ 

where ytgt is a test score for student i in grade g in year /, U 2 t and ^ 2 g are year and grade effects, 
Xi is a vector of pre-lottery demographic characteristics (race, special education, limited English 
proficiency, subsidized lunch status, and a female-minority interaction), and represents 
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random fluctuations in test scores. The set of includes a separate dummy variable for every 
combination of observed charter school lotteries (indexed by /) entered by students in the lottery 
sample. The variable of interest, measures years spent in charter schools between 

application and test dates. The parameter t captures the causal effect of charter school 
attendance. 

Ordinary Least Squares (OLS) estimates of this equation may be biased because students do not 
choose to attend charter schools randomly. We therefore use a dummy variable, Zj, indicating 
lottery offers as an instrument for time spent in charter school. This procedure isolates the 
random variation in charter school attendance produced by entrance lotteries and generates 
unbiased estimates of the causal effect of interest. We produce 2SLS estimates separately for 
urban and non-urban schools. 

Given estimates of the causal effect of charter school attendance, we decompose differences in 
the effects of urban and non-urban schools into student-level and school-level factors. First, we 
examine levels of test scores in and out of charter schools for urban and non-urban charter 
applicants. Second, we perform a Blinder-Oaxaca (1973) decomposition of the urban charter 
advantage to determine the portion of the difference in effectiveness due to student 
demographics. Finally, we analyze the relationship between school-level causal effects and 
policies. For this analysis, we produce estimates using observational methods, which allows us 
to compare lottery and non-lottery charter schools in addition to urban and non-urban ones. The 
school-level analysis uses equations of the form 

fs^xpo + c;,xpt + 

Here fg is an estimate of the causal effect of school S and Cg is a vector of school characteristics 
and policies measured in our survey of administrators. The results of this analysis reveal the 
policies that are most strongly associated with charter school effectiveness. 

Data Collection and Analysis: 

Description of the methods for coiiecting and anaiyzing data. 

We attempted to collect lottery data for the set of Massachusetts charter schools serving middle 
and high school grades with entrance lotteries in grades 4 through 10. We collected lists of 
lottery applicants, winners, and losers from each charter school with available records. These 
records were matched by name, grade, and year of application to an administrative database of 
student demographics, enrollment, and test scores from Massachusetts’ Student Information 
Management System (SIMS) and Comprehensive Assessment System (MCAS). These 
databases were obtained through a data-use agreement with the Massachusetts Department of 
Elementary and Secondary Education (DESE). The match rates from the lottery data to the 
administrative data were 92 percent for middle school and 94 percent for high school. 

Findings/ Results: 

Description of the main findings with specific detaiis. 
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Our results suggest that urban charter schools in Massachusetts have much larger effects on 
student test scores than non-urban charter schools. Table 3 shows 2SLS estimates of the per-year 
effects of charter school. The estimates, which are in standard deviation units (cr), are reported 
separately by urban status and school level. In middle school, urban charter attendance increases 
ELA scores by 0.14cr per year and math scores by 0.34cr per year. In contrast, attendance at non- 
urban charter middle schools reduces both ELA and math scores by 0. 16cr per year. All of the 
middle school estimates are highly statistically significant. The estimates for high school are 
similar, though results for non-urban high schools are imprecise due to small sample sizes. 

Our analysis of charter school effect heterogeneity shows that the striking difference between the 
achievement effects of urban and non-urban charter schools is only partially explained by 
student-level factors. Compared to non-urban students, urban charter students start out at low 
levels of achievement, though their non-charter scores are typical of the general population of 
urban students. Charter attendance raises the scores of urban students to levels comparable to 
their non-urban counterparts, while scores for non-urban students are largely unchanged or 
slightly reduced by charter attendance. Our Blinder-Oaxaca analysis shows that student 
demographics explain some of the urban advantage, as urban schools are most effective for low- 
achieving minority students (which are overrepresented in urban areas), but differences in 
effectiveness within demographic groups are large as well. 

Einally, our school-level analysis suggests that the No instructional approach can 

explain the effectiveness of urban schools relative to non-urban ones, as well as the difference in 
effectiveness between schools with and without entrance lotteries. As shown in Table 4, 
controlling for No ExCUSes status accounts for both the urban and lottery advantages; these 
differences are small and statistically insignificant in our school-level regressions conditional on 
No Excuses. Other important characteristics, like time in school and per -pupil expenditure, are 
not individually associated with charter school effectiveness and do not explain the No ExCUSes 
advantage. 

Conclusions: 

Description of conciusions, recommendations, and i imitations based on findings. 

Our analysis demonstrates that urban charter schools in Massachusetts dramatically increase 
student test scores, while non-urban charter schools appear to be largely ineffective and may 
reduce achievement for some. Candidate explanations for this constellation of findings include 
the fact that urban charter schools serve larger shares of minority students in districts where the 
surrounding achievement level is generally low, keep their students in school longer, spend more 
money per-pupil, and are much more likely to identify with the No ExCUSes instructional 
approach than are non-urban schools. Our analysis examines the contribution of these student- 
and school-level factors to the urban charter advantage. The results suggest that student-level 
factors only partially explain our findings, while adherence to the No ExCUSes approach accounts 
for the urban and lottery-sample advantages. In ongoing work, we're looking at a variety of post- 
secondary outcomes in an effort to determine whether the heterogeneous findings for 
achievement have longer-term consequences. We also hope to investigate the effectiveness of 
No Excuses education for non-urban students by drawing new samples of students and schools 
from other states. 
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Table 1: Descriptive Statistics 





Regular Public Schools 


Charter schools (eligible) 


Charter applicants (lottery) 




Urban 

0) 


Non-urban 

(2) 


Urban 

(3) 


Non-urban 

(4) 


Urban 

(5) 


Non-urban 

(6) 






Panel A. Middle Schools (5th-8th grade) 




Female 


0.486 


0.488 


0.498 


0.476 


0.496 


0.510 


Black 


0.185 


0.027 


0.407 


0.036 


0.479 


0.022 


Hispanic 


0.314 


0.036 


0.237 


0.048 


0.234 


0.025 


Special education 


0.190 


0.163 


0.166 


0.160 


0.176 


0.184 


Subsidized lunch 


0.681 


0.141 


0.650 


0.216 


0.686 


0.102 


Limited English proficiency 


0.150 


0.016 


0.077 


0.025 


0.086 


0.008 


Baseline Math score 


-0.430 


0.213 


-0.339 


0.239 


-0.352 


0.306 


Baseline ELA score 


-0.464 


0.234 


-0.330 


0.261 


-0.373 


0.392 


Years in charter 


0.000 


0.000 


2.027 


1.960 


1.341 


1.002 


N (students) 


153374 


369866 


6625 


8316 


4126 


1693 


N (schools) 


262 


390 


16 


11 


9 


7 








Panel B. High Schools (1 0th grade) 




Female 


0.500 


0.494 


0.555 


0.549 


0.549 


0.539 


Black 


0.190 


0.028 


0.535 


0.020 


0.615 


0.029 


Hispanic 


0.272 


0.032 


0.176 


0.010 


0.256 


0.017 


Special education 


0.169 


0.155 


0.160 


0.105 


0.174 


0.115 


Subsidized lunch 


0.606 


0.122 


0.600 


0.146 


0.716 


0.120 


Limited English proficiency 


0.093 


0.009 


0.022 


0.005 


0.035 


0.003 


Baseline Math score 


-0.420 


0.271 


-0.413 


0.322 


-0.315 


0.445 


Baseline ELA score 


-0.387 


0.282 


-0.325 


0.413 


-0.306 


0.562 


Years in charter 


0.000 


0.000 


1.765 


1.797 


0.627 


1.292 


N (students) 


116593 


313366 


2198 


783 


2973 


349 


N (schools) 


101 


304 


8 


2 


4 


2 



Notes: This table reports descriptive statistics for the sample of pubhc school students (columns 1 and 2), the sample of students in 
eligible charter schools (columns 3 and 4), and the sample of charter applicants (columns 5 and 6) from 2002-2010. The sample is 
restricted to students in Massachusetts public schools at baseline with at least one followup test score. The number of schools in 
columns (1) and (2) is counted in 6th grade for middle school and 10th grade for high school. Years in charter school is measured 
through 8th grade for middle school and 10th grade for high school. 
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Table 2: Characteristics of Charter Schools 





Statewide 


Urban 


Non-urban 


Characteristic 


(1) 


(2) 


(3) 



Years open 
Days per year 
Average minutes per day 
Have Saturday school 
Avg. math instruction (min) 

Avg. reading instruction (min) 

CMO or Network Affiliation 
Fully or somewhat "No excuses" 

Parent contract 
Student contract 
Uniforms 

Reward and punishment system 
Avg. per-pupil expenditure 
Tide I eligible 
Number of teachers 
Studentiteacher ratio 
Licensed teachers 
Proportion 32 and younger 
Proportion 49 and older 
Left voluntarily 
Left involuntarily 

Require staff to take calls after hours 
Unpaid tutors/volunteers 
Paid tutors 

N (schools) 



9.43 


8.18 


11.36 


186.18 


188.53 


182.55 


447.86 


464.35 


422.36 


0.321 


0.471 


0.091 


80.93 


94.56 


59.86 


84.00 


99.62 


59.86 


0.357 


0.294 


0.455 


0.429 


0.706 


0.000 


0.679 


0.824 


0.455 


0.643 


0.706 


0.545 


0.821 


0.882 


0.727 


0.464 


0.647 


0.182 


12824.19 


14095.53 


11090.55 


0.857 


1.000 


0.636 


25.736 


22.735 


30.373 


11.614 


11.565 


11.691 


51.146 


51.853 


50.055 


0.577 


0.709 


0.384 


0.129 


0.058 


0.233 


2.278 


1.969 


2.727 


1.296 


1.500 


1.000 


0.071 


0.059 


0.091 


0.786 


0.706 


0.909 


0.143 


0.235 


0.000 


28 


17 


11 



Notes: This table reports results from a survey of Massachusetts charter schools with enty in middle (4th- 
7th) or high school (9th) grades. The survey sample excludes schools closed prior to 2010, schools that 
were not open before Fall 2010, and schools serving non-traditional student populations. Twenty-eight of 
32 ehgible schools responded to the survey. 
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Table 3: Lotteiy Resiilte for Urban and Non-urban Charter Schools 











Urban 






Non-urban 




School level 


Subject 




First Stage 
(1) 


Reduced Form 
(2) 


2SLS 

(3) 


First Stage 
(4) 


Reduced Form 

(5) 


2SLS 

(6) 


Middle 


ELA 


N 


1.001*** 

(0.055) 


0.141*** 

(0.035) 

8762 


0.140*** 

(0.033) 


0.978*** 

(0.081) 


-0.155*** 

(0.045) 

3364 


-0 156*** 
(0.045) 




Math 


N 


0.990*** 

(0.054) 


0.333*** 

(0.038) 

9015 


0.336*** 

(0.036) 


0.996*** 

(0.081) 


-0.159*** 

(0.050) 

3331 


-0.155*** 

(0.051) 


High 


ELA 


N 


0,494 *** 
(0.105) 


0.117** 

(0.051) 

2954 


0.236*** 

(0.079) 


1.082*** 

(0.153) 


-0.014 

(0.116) 

349 


-0.009 

(0.105) 




Math 


N 


0.495 *** 
(0.105) 


0.178*** 

(0.066) 

2910 


Q 359*** 

(0.092) 


1.088*** 

(0.158) 


-0.274* 

(0.162) 

345 


-0.246* 

(0.148) 




Writing Topic 


N 


0.500*** 

(0.105) 


0.166*** 

(0.058) 

2920 


0.332*** 

(0.090) 


1.082*** 

(0.153) 


-0.157 

(0.222) 

348 


-0.139 

(0.204) 




Writing Composition 
N 


0.500*** 

(0.105) 


0.149** 

(0.060) 

2920 


0.298*** 

(0.096) 


1.082*** 

(0.153) 


-0.155 

(0.213) 

348 


-0.137 

(0.196) 



Notes: This table reports estimates of the effects of years in urban and non-urban charter schools on test scores. The sample is restricted to students 
with baseline demographic characteristics who attended a Massachusetts public school when tested, and excludes students with sibling priority and late 
applicants. Estimates are produced by a 2SLS procedure using urban and non-urban lottery offers as instruments for attendance at urban and non- 
urban charter schools. All models control for race, sex, special education, limited English proficiency, subsidized lunch status, and a female by 
minority dummy. Year of birth, year of test, and risk set dummies are also included. Middle school regressions pool post-lottery outcomes from 4th 
through 8th grade and cluster by student identifier as well as school-grade-year. High school regressions include only scores for 10th grade, and cluster 
by school-grade-year. 

*significant at 10%; **significant at 5%; ***significant at 1% 
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Table 4: Effects of School Characteristics 







Math 






ELA 






V ariable 


(1) 


(2) 


(3) 


(4) 


(5) 


(6) 


(7) 


(8) 


Urban 


0.198*** 


0.072 


0.008 


-0.041 


0.120*** 


0.062* 


0.011 


0.014 




(0.057) 


(0.082) 


(0.062) 


(0.053) 


(0.036) 


(0.037) 


(0.033) 


(0.042) 


Total minutes per day/100 


- 


0.154* 




0.095 


- 


0.080* 


- 


0.055 






(0.090) 




(0.078) 




(0.042) 




(0.038) 


Minutes in relevant subject/100 


- 


0.203 


- 


0.207 


- 


0.023 


- 


0.007 






(0.211) 




(0.168) 




(0.075) 




(0.068) 


Per-pupil expenditure/1000 


- 


-0.002 


- 


-0.009 


- 


0.004 


- 


-0.001 






(0.014) 




(0.010) 




(0.008) 




(0.009) 


School is No Excuses 


- 


- 


0.306*** 


0.231*** 


- 


- 


0.169*** 


0.117** 








(0.082) 


(0.060) 






(0.045) 


(0.048) 


Lottery 


0.154** 


0.086* 


0.051 


0.038 


0.101** 


0.055 


0.047 


0.033 




(0.069) 


(0.050) 


(0.052) 


(0.041) 


(0.043) 


(0.035) 


(0.036) 


(0.033) 


High School 


0.039 


0.078 


0.035 


0.087 


0.069* 


0.076* 


0.062* 


0.078* 




(0.071) 


(0.065) 


(0.052) 


(0.057) 


(0.036) 


(0.040) 


(0.032) 


(0.040) 


Constant 


-0.131* 


-0.835** 


-0.064 


-0.490 


-0.085* 


-0.445** 


-0.047 


-0.267 




(0.067) 


(0.375) 


(0.043) 


(0.299) 


(0.045) 


(0.176) 


(0.033) 


(0.183) 


N 


30 


28 


30 


28 


30 


28 


30 


28 



Notes: This table reports regressions of school-specific treatment effects on school characteristics. The sample includes onfy schools that completed the charter 
survey. Regressions weight by the inverse of the standard error of the coefficient estimates and cluster at the school level. 

‘'‘significant at 10^ ‘'“'‘significant at 5%; ’'“'“'‘significant at 1% 
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